<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<!--[if IE]><meta http-equiv="X-UA-Compatible" content="IE=edge"><![endif]-->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 1.5.5.dev">
<title>流式计算概述</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700">
<style>
/* Asciidoctor default stylesheet | MIT License | http://asciidoctor.org */
/* Remove comment around @import statement below when using as a custom stylesheet */
/*@import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700";*/
article,aside,details,figcaption,figure,footer,header,hgroup,main,nav,section,summary{display:block}
audio,canvas,video{display:inline-block}
audio:not([controls]){display:none;height:0}
[hidden],template{display:none}
script{display:none!important}
html{font-family:sans-serif;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}
body{margin:0}
a{background:transparent}
a:focus{outline:thin dotted}
a:active,a:hover{outline:0}
h1{font-size:2em;margin:.67em 0}
abbr[title]{border-bottom:1px dotted}
b,strong{font-weight:bold}
dfn{font-style:italic}
hr{-moz-box-sizing:content-box;box-sizing:content-box;height:0}
mark{background:#ff0;color:#000}
code,kbd,pre,samp{font-family:monospace;font-size:1em}
pre{white-space:pre-wrap}
q{quotes:"\201C" "\201D" "\2018" "\2019"}
small{font-size:80%}
sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}
sup{top:-.5em}
sub{bottom:-.25em}
img{border:0}
svg:not(:root){overflow:hidden}
figure{margin:0}
fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}
legend{border:0;padding:0}
button,input,select,textarea{font-family:inherit;font-size:100%;margin:0}
button,input{line-height:normal}
button,select{text-transform:none}
button,html input[type="button"],input[type="reset"],input[type="submit"]{-webkit-appearance:button;cursor:pointer}
button[disabled],html input[disabled]{cursor:default}
input[type="checkbox"],input[type="radio"]{box-sizing:border-box;padding:0}
input[type="search"]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}
input[type="search"]::-webkit-search-cancel-button,input[type="search"]::-webkit-search-decoration{-webkit-appearance:none}
button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}
textarea{overflow:auto;vertical-align:top}
table{border-collapse:collapse;border-spacing:0}
*,*:before,*:after{-moz-box-sizing:border-box;-webkit-box-sizing:border-box;box-sizing:border-box}
html,body{font-size:100%}
body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;font-weight:400;font-style:normal;line-height:1;position:relative;cursor:auto}
a:hover{cursor:pointer}
img,object,embed{max-width:100%;height:auto}
object,embed{height:100%}
img{-ms-interpolation-mode:bicubic}
.left{float:left!important}
.right{float:right!important}
.text-left{text-align:left!important}
.text-right{text-align:right!important}
.text-center{text-align:center!important}
.text-justify{text-align:justify!important}
.hide{display:none}
body{-webkit-font-smoothing:antialiased}
img,object,svg{display:inline-block;vertical-align:middle}
textarea{height:auto;min-height:50px}
select{width:100%}
.center{margin-left:auto;margin-right:auto}
.spread{width:100%}
p.lead,.paragraph.lead>p,#preamble>.sectionbody>.paragraph:first-of-type p{font-size:1.21875em;line-height:1.6}
.subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em}
div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0;direction:ltr}
a{color:#2156a5;text-decoration:underline;line-height:inherit}
a:hover,a:focus{color:#1d4b8f}
a img{border:none}
p{font-family:inherit;font-weight:400;font-size:1em;line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility}
p aside{font-size:.875em;line-height:1.35;font-style:italic}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em}
h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0}
h1{font-size:2.125em}
h2{font-size:1.6875em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em}
h4,h5{font-size:1.125em}
h6{font-size:1em}
hr{border:solid #ddddd8;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em;height:0}
em,i{font-style:italic;line-height:inherit}
strong,b{font-weight:bold;line-height:inherit}
small{font-size:60%;line-height:inherit}
code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)}
ul,ol,dl{font-size:1em;line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit}
ul,ol,ul.no-bullet,ol.no-bullet{margin-left:1.5em}
ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0;font-size:1em}
ul.square li ul,ul.circle li ul,ul.disc li ul{list-style:inherit}
ul.square{list-style-type:square}
ul.circle{list-style-type:circle}
ul.disc{list-style-type:disc}
ul.no-bullet{list-style:none}
ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0}
dl dt{margin-bottom:.3125em;font-weight:bold}
dl dd{margin-bottom:1.25em}
abbr,acronym{text-transform:uppercase;font-size:90%;color:rgba(0,0,0,.8);border-bottom:1px dotted #ddd;cursor:help}
abbr{text-transform:none}
blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd}
blockquote cite{display:block;font-size:.9375em;color:rgba(0,0,0,.6)}
blockquote cite:before{content:"\2014 \0020"}
blockquote cite a,blockquote cite a:visited{color:rgba(0,0,0,.6)}
blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)}
@media only screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2}
    h1{font-size:2.75em}
    h2{font-size:2.3125em}
    h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em}
    h4{font-size:1.4375em}}
table{background:#fff;margin-bottom:1.25em;border:solid 1px #dedede}
table thead,table tfoot{background:#f7f8f7;font-weight:bold}
table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left}
table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)}
table tr.even,table tr.alt,table tr:nth-of-type(even){background:#f8f8f7}
table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{display:table-cell;line-height:1.6}
body{tab-size:4}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em}
h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400}
.clearfix:before,.clearfix:after,.float-group:before,.float-group:after{content:" ";display:table}
.clearfix:after,.float-group:after{clear:both}
*:not(pre)>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background-color:#f7f7f8;-webkit-border-radius:4px;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed}
pre,pre>code{line-height:1.45;color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;text-rendering:optimizeSpeed}
.keyseq{color:rgba(51,51,51,.8)}
kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background-color:#f7f7f7;border:1px solid #ccc;-webkit-border-radius:3px;border-radius:3px;-webkit-box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em white inset;box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em #fff inset;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap}
.keyseq kbd:first-child{margin-left:0}
.keyseq kbd:last-child{margin-right:0}
.menuseq,.menu{color:rgba(0,0,0,.8)}
b.button:before,b.button:after{position:relative;top:-1px;font-weight:400}
b.button:before{content:"[";padding:0 3px 0 2px}
b.button:after{content:"]";padding:0 2px 0 3px}
p a>code:hover{color:rgba(0,0,0,.9)}
#header,#content,#footnotes,#footer{width:100%;margin-left:auto;margin-right:auto;margin-top:0;margin-bottom:0;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em}
#header:before,#header:after,#content:before,#content:after,#footnotes:before,#footnotes:after,#footer:before,#footer:after{content:" ";display:table}
#header:after,#content:after,#footnotes:after,#footer:after{clear:both}
#content{margin-top:1.25em}
#content:before{content:none}
#header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0}
#header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #ddddd8}
#header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #ddddd8;padding-bottom:8px}
#header .details{border-bottom:1px solid #ddddd8;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:-ms-flexbox;display:-webkit-flex;display:flex;-ms-flex-flow:row wrap;-webkit-flex-flow:row wrap;flex-flow:row wrap}
#header .details span:first-child{margin-left:-.125em}
#header .details span.email a{color:rgba(0,0,0,.85)}
#header .details br{display:none}
#header .details br+span:before{content:"\00a0\2013\00a0"}
#header .details br+span.author:before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)}
#header .details br+span#revremark:before{content:"\00a0|\00a0"}
#header #revnumber{text-transform:capitalize}
#header #revnumber:after{content:"\00a0"}
#content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #ddddd8;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem}
#toc{border-bottom:1px solid #efefed;padding-bottom:.5em}
#toc>ul{margin-left:.125em}
#toc ul.sectlevel0>li>a{font-style:italic}
#toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0}
#toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none}
#toc li{line-height:1.3334;margin-top:.3334em}
#toc a{text-decoration:none}
#toc a:active{text-decoration:underline}
#toctitle{color:#7a2518;font-size:1.2em}
@media only screen and (min-width:768px){#toctitle{font-size:1.375em}
    body.toc2{padding-left:15em;padding-right:0}
    #toc.toc2{margin-top:0!important;background-color:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #efefed;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto}
    #toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em}
    #toc.toc2>ul{font-size:.9em;margin-bottom:0}
    #toc.toc2 ul ul{margin-left:0;padding-left:1em}
    #toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em}
    body.toc2.toc-right{padding-left:0;padding-right:15em}
    body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #efefed;left:auto;right:0}}
@media only screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0}
    #toc.toc2{width:20em}
    #toc.toc2 #toctitle{font-size:1.375em}
    #toc.toc2>ul{font-size:.95em}
    #toc.toc2 ul ul{padding-left:1.25em}
    body.toc2.toc-right{padding-left:0;padding-right:20em}}
#content #toc{border-style:solid;border-width:1px;border-color:#e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;-webkit-border-radius:4px;border-radius:4px}
#content #toc>:first-child{margin-top:0}
#content #toc>:last-child{margin-bottom:0}
#footer{max-width:100%;background-color:rgba(0,0,0,.8);padding:1.25em}
#footer-text{color:rgba(255,255,255,.8);line-height:1.44}
.sect1{padding-bottom:.625em}
@media only screen and (min-width:768px){.sect1{padding-bottom:1.25em}}
.sect1+.sect1{border-top:1px solid #efefed}
#content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400}
#content h1>a.anchor:before,h2>a.anchor:before,h3>a.anchor:before,#toctitle>a.anchor:before,.sidebarblock>.content>.title>a.anchor:before,h4>a.anchor:before,h5>a.anchor:before,h6>a.anchor:before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em}
#content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible}
#content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none}
#content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221}
.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em}
.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic}
table.tableblock>caption.title{white-space:nowrap;overflow:visible;max-width:0}
.paragraph.lead>p,#preamble>.sectionbody>.paragraph:first-of-type p{color:rgba(0,0,0,.85)}
table.tableblock #preamble>.sectionbody>.paragraph:first-of-type p{font-size:inherit}
.admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%}
.admonitionblock>table td.icon{text-align:center;width:80px}
.admonitionblock>table td.icon img{max-width:none}
.admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase}
.admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #ddddd8;color:rgba(0,0,0,.6)}
.admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0}
.exampleblock>.content{border-style:solid;border-width:1px;border-color:#e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;-webkit-border-radius:4px;border-radius:4px}
.exampleblock>.content>:first-child{margin-top:0}
.exampleblock>.content>:last-child{margin-bottom:0}
.sidebarblock{border-style:solid;border-width:1px;border-color:#e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;-webkit-border-radius:4px;border-radius:4px}
.sidebarblock>:first-child{margin-top:0}
.sidebarblock>:last-child{margin-bottom:0}
.sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center}
.exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0}
.literalblock pre,.listingblock pre:not(.highlight),.listingblock pre[class="highlight"],.listingblock pre[class^="highlight "],.listingblock pre.CodeRay,.listingblock pre.prettyprint{background:#f7f7f8}
.sidebarblock .literalblock pre,.sidebarblock .listingblock pre:not(.highlight),.sidebarblock .listingblock pre[class="highlight"],.sidebarblock .listingblock pre[class^="highlight "],.sidebarblock .listingblock pre.CodeRay,.sidebarblock .listingblock pre.prettyprint{background:#f2f1f1}
.literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{-webkit-border-radius:4px;border-radius:4px;word-wrap:break-word;padding:1em;font-size:.8125em}
.literalblock pre.nowrap,.literalblock pre[class].nowrap,.listingblock pre.nowrap,.listingblock pre[class].nowrap{overflow-x:auto;white-space:pre;word-wrap:normal}
@media only screen and (min-width:768px){.literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{font-size:.90625em}}
@media only screen and (min-width:1280px){.literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{font-size:1em}}
.literalblock.output pre{color:#f7f7f8;background-color:rgba(0,0,0,.9)}
.listingblock pre.highlightjs{padding:0}
.listingblock pre.highlightjs>code{padding:1em;-webkit-border-radius:4px;border-radius:4px}
.listingblock pre.prettyprint{border-width:0}
.listingblock>.content{position:relative}
.listingblock code[data-lang]:before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:#999}
.listingblock:hover code[data-lang]:before{display:block}
.listingblock.terminal pre .command:before{content:attr(data-prompt);padding-right:.5em;color:#999}
.listingblock.terminal pre .command:not([data-prompt]):before{content:"$"}
table.pyhltable{border-collapse:separate;border:0;margin-bottom:0;background:none}
table.pyhltable td{vertical-align:top;padding-top:0;padding-bottom:0;line-height:1.45}
table.pyhltable td.code{padding-left:.75em;padding-right:0}
pre.pygments .lineno,table.pyhltable td:not(.code){color:#999;padding-left:0;padding-right:.5em;border-right:1px solid #ddddd8}
pre.pygments .lineno{display:inline-block;margin-right:.25em}
table.pyhltable .linenodiv{background:none!important;padding-right:0!important}
.quoteblock{margin:0 1em 1.25em 1.5em;display:table}
.quoteblock>.title{margin-left:-1.5em;margin-bottom:.75em}
.quoteblock blockquote,.quoteblock blockquote p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify}
.quoteblock blockquote{margin:0;padding:0;border:0}
.quoteblock blockquote:before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)}
.quoteblock blockquote>.paragraph:last-child p{margin-bottom:0}
.quoteblock .attribution{margin-top:.5em;margin-right:.5ex;text-align:right}
.quoteblock .quoteblock{margin-left:0;margin-right:0;padding:.5em 0;border-left:3px solid rgba(0,0,0,.6)}
.quoteblock .quoteblock blockquote{padding:0 0 0 .75em}
.quoteblock .quoteblock blockquote:before{display:none}
.verseblock{margin:0 1em 1.25em 1em}
.verseblock pre{font-family:"Open Sans","DejaVu Sans",sans;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility}
.verseblock pre strong{font-weight:400}
.verseblock .attribution{margin-top:1.25rem;margin-left:.5ex}
.quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic}
.quoteblock .attribution br,.verseblock .attribution br{display:none}
.quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)}
.quoteblock.abstract{margin:0 0 1.25em 0;display:block}
.quoteblock.abstract blockquote,.quoteblock.abstract blockquote p{text-align:left;word-spacing:0}
.quoteblock.abstract blockquote:before,.quoteblock.abstract blockquote p:first-of-type:before{display:none}
table.tableblock{max-width:100%;border-collapse:separate}
table.tableblock td>.paragraph:last-child p>p:last-child,table.tableblock th>p:last-child,table.tableblock td>p:last-child{margin-bottom:0}
table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede}
table.grid-all th.tableblock,table.grid-all td.tableblock{border-width:0 1px 1px 0}
table.grid-all tfoot>tr>th.tableblock,table.grid-all tfoot>tr>td.tableblock{border-width:1px 1px 0 0}
table.grid-cols th.tableblock,table.grid-cols td.tableblock{border-width:0 1px 0 0}
table.grid-all *>tr>.tableblock:last-child,table.grid-cols *>tr>.tableblock:last-child{border-right-width:0}
table.grid-rows th.tableblock,table.grid-rows td.tableblock{border-width:0 0 1px 0}
table.grid-all tbody>tr:last-child>th.tableblock,table.grid-all tbody>tr:last-child>td.tableblock,table.grid-all thead:last-child>tr>th.tableblock,table.grid-rows tbody>tr:last-child>th.tableblock,table.grid-rows tbody>tr:last-child>td.tableblock,table.grid-rows thead:last-child>tr>th.tableblock{border-bottom-width:0}
table.grid-rows tfoot>tr>th.tableblock,table.grid-rows tfoot>tr>td.tableblock{border-width:1px 0 0 0}
table.frame-all{border-width:1px}
table.frame-sides{border-width:0 1px}
table.frame-topbot{border-width:1px 0}
th.halign-left,td.halign-left{text-align:left}
th.halign-right,td.halign-right{text-align:right}
th.halign-center,td.halign-center{text-align:center}
th.valign-top,td.valign-top{vertical-align:top}
th.valign-bottom,td.valign-bottom{vertical-align:bottom}
th.valign-middle,td.valign-middle{vertical-align:middle}
table thead th,table tfoot th{font-weight:bold}
tbody tr th{display:table-cell;line-height:1.6;background:#f7f8f7}
tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold}
p.tableblock>code:only-child{background:none;padding:0}
p.tableblock{font-size:1em}
td>div.verse{white-space:pre}
ol{margin-left:1.75em}
ul li ol{margin-left:1.5em}
dl dd{margin-left:1.125em}
dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0}
ol>li p,ul>li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em}
ul.unstyled,ol.unnumbered,ul.checklist,ul.none{list-style-type:none}
ul.unstyled,ol.unnumbered,ul.checklist{margin-left:.625em}
ul.checklist li>p:first-child>.fa-square-o:first-child,ul.checklist li>p:first-child>.fa-check-square-o:first-child{width:1em;font-size:.85em}
ul.checklist li>p:first-child>input[type="checkbox"]:first-child{width:1em;position:relative;top:1px}
ul.inline{margin:0 auto .625em auto;margin-left:-1.375em;margin-right:0;padding:0;list-style:none;overflow:hidden}
ul.inline>li{list-style:none;float:left;margin-left:1.375em;display:block}
ul.inline>li>*{display:block}
.unstyled dl dt{font-weight:400;font-style:normal}
ol.arabic{list-style-type:decimal}
ol.decimal{list-style-type:decimal-leading-zero}
ol.loweralpha{list-style-type:lower-alpha}
ol.upperalpha{list-style-type:upper-alpha}
ol.lowerroman{list-style-type:lower-roman}
ol.upperroman{list-style-type:upper-roman}
ol.lowergreek{list-style-type:lower-greek}
.hdlist>table,.colist>table{border:0;background:none}
.hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none}
td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em}
td.hdlist1{font-weight:bold;padding-bottom:1.25em}
.literalblock+.colist,.listingblock+.colist{margin-top:-.5em}
.colist>table tr>td:first-of-type{padding:0 .75em;line-height:1}
.colist>table tr>td:last-of-type{padding:.25em 0}
.thumb,.th{line-height:0;display:inline-block;border:solid 4px #fff;-webkit-box-shadow:0 0 0 1px #ddd;box-shadow:0 0 0 1px #ddd}
.imageblock.left,.imageblock[style*="float: left"]{margin:.25em .625em 1.25em 0}
.imageblock.right,.imageblock[style*="float: right"]{margin:.25em 0 1.25em .625em}
.imageblock>.title{margin-bottom:0}
.imageblock.thumb,.imageblock.th{border-width:6px}
.imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em}
.image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0}
.image.left{margin-right:.625em}
.image.right{margin-left:.625em}
a.image{text-decoration:none;display:inline-block}
a.image object{pointer-events:none}
sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super}
sup.footnote a,sup.footnoteref a{text-decoration:none}
sup.footnote a:active,sup.footnoteref a:active{text-decoration:underline}
#footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em}
#footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em 0;border-width:1px 0 0 0}
#footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;text-indent:-1.05em;margin-bottom:.2em}
#footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none}
#footnotes .footnote:last-of-type{margin-bottom:0}
#content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0}
.gist .file-data>table{border:0;background:#fff;width:100%;margin-bottom:0}
.gist .file-data>table td.line-data{width:99%}
div.unbreakable{page-break-inside:avoid}
.big{font-size:larger}
.small{font-size:smaller}
.underline{text-decoration:underline}
.overline{text-decoration:overline}
.line-through{text-decoration:line-through}
.aqua{color:#00bfbf}
.aqua-background{background-color:#00fafa}
.black{color:#000}
.black-background{background-color:#000}
.blue{color:#0000bf}
.blue-background{background-color:#0000fa}
.fuchsia{color:#bf00bf}
.fuchsia-background{background-color:#fa00fa}
.gray{color:#606060}
.gray-background{background-color:#7d7d7d}
.green{color:#006000}
.green-background{background-color:#007d00}
.lime{color:#00bf00}
.lime-background{background-color:#00fa00}
.maroon{color:#600000}
.maroon-background{background-color:#7d0000}
.navy{color:#000060}
.navy-background{background-color:#00007d}
.olive{color:#606000}
.olive-background{background-color:#7d7d00}
.purple{color:#600060}
.purple-background{background-color:#7d007d}
.red{color:#bf0000}
.red-background{background-color:#fa0000}
.silver{color:#909090}
.silver-background{background-color:#bcbcbc}
.teal{color:#006060}
.teal-background{background-color:#007d7d}
.white{color:#bfbfbf}
.white-background{background-color:#fafafa}
.yellow{color:#bfbf00}
.yellow-background{background-color:#fafa00}
span.icon>.fa{cursor:default}
.admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default}
.admonitionblock td.icon .icon-note:before{content:"\f05a";color:#19407c}
.admonitionblock td.icon .icon-tip:before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111}
.admonitionblock td.icon .icon-warning:before{content:"\f071";color:#bf6900}
.admonitionblock td.icon .icon-caution:before{content:"\f06d";color:#bf3400}
.admonitionblock td.icon .icon-important:before{content:"\f06a";color:#bf0000}
.conum[data-value]{display:inline-block;color:#fff!important;background-color:rgba(0,0,0,.8);-webkit-border-radius:100px;border-radius:100px;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold}
.conum[data-value] *{color:#fff!important}
.conum[data-value]+b{display:none}
.conum[data-value]:after{content:attr(data-value)}
pre .conum[data-value]{position:relative;top:-.125em}
b.conum *{color:inherit!important}
.conum:not([data-value]):empty{display:none}
dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility}
h1,h2,p,td.content,span.alt{letter-spacing:-.01em}
p strong,td.content strong,div.footnote strong{letter-spacing:-.005em}
p,blockquote,dt,td.content,span.alt{font-size:1.0625rem}
p{margin-bottom:1.25rem}
.sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em}
.exampleblock>.content{background-color:#fffef7;border-color:#e0e0dc;-webkit-box-shadow:0 1px 4px #e0e0dc;box-shadow:0 1px 4px #e0e0dc}
.print-only{display:none!important}
@media print{@page{margin:1.25cm .75cm}
    *{-webkit-box-shadow:none!important;box-shadow:none!important;text-shadow:none!important}
    a{color:inherit!important;text-decoration:underline!important}
    a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important}
    a[href^="http:"]:not(.bare):after,a[href^="https:"]:not(.bare):after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em}
    abbr[title]:after{content:" (" attr(title) ")"}
    pre,blockquote,tr,img,object,svg{page-break-inside:avoid}
    thead{display:table-header-group}
    svg{max-width:100%}
    p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3}
    h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid}
    #toc,.sidebarblock,.exampleblock>.content{background:none!important}
    #toc{border-bottom:1px solid #ddddd8!important;padding-bottom:0!important}
    .sect1{padding-bottom:0!important}
    .sect1+.sect1{border:0!important}
    #header>h1:first-child{margin-top:1.25rem}
    body.book #header{text-align:center}
    body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em 0}
    body.book #header .details{border:0!important;display:block;padding:0!important}
    body.book #header .details span:first-child{margin-left:0!important}
    body.book #header .details br{display:block}
    body.book #header .details br+span:before{content:none!important}
    body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important}
    body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always}
    .listingblock code[data-lang]:before{display:block}
    #footer{background:none!important;padding:0 .9375em}
    #footer-text{color:rgba(0,0,0,.6)!important;font-size:.9em}
    .hide-on-print{display:none!important}
    .print-only{display:block!important}
    .hide-for-print{display:none!important}
    .show-for-print{display:inherit!important}}
</style>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.6.3/css/font-awesome.min.css">
</head>
<body class="article">
<div id="header">
<h1>流式计算概述</h1>
</div>
<div id="content">
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p>我们日常开发的系统，可以按照发起请求 &#8594; 收到响应的时间分为三类：</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Services (online systems)</p>
</div>
<div class="paragraph">
<p>服务等待用户请求或指令到达。当收到请求或指令时，服务试图尽可能快地处理它，并发回一个响应。响应时间通常是服务性能的主要衡量指标。而可用性同样非常重要(如果客户端无法访问服务，用户可能会收到一个报错信息)。</p>
</div>
<div class="paragraph">
<p>Batch processing systems (offline systems)</p>
</div>
<div class="paragraph">
<p>批处理系统接收大量输入数据，运行一个作业来处理数据，并产生输出数据。作业往往需要执行一段时间(从几分钟到几天)，所以用户通常不会等待作业完成。相反，批量作业通常会定期运行(例如每天一次)。批处理作业的主要性能衡量标准通常是吞吐量(处理一定大小的输入数据集所需的时间)。</p>
</div>
<div class="paragraph">
<p>Stream processing systems (near-real-time systems)</p>
</div>
<div class="paragraph">
<p>流处理系统介于在线与离线/批处理之间(所以有时称为近实时或近线处理)。与批处理系统类似，流处理系统处理输入并产生输出(而不是响应请求)。但是，流式作业在事件发生后不久即可对事件进行处理，而批处理作业则使用固定的一组输入数据进行操作。这种差异使得流处理系统比批处理系统具有更低的延迟。<strong>流处理是在批处理的基础上进行的。</strong></p>
</div>
</blockquote>
<div class="attribution">
&#8212; Designing Data-intensive applications
</div>
</div>
<div class="paragraph">
<p>map reduce 基本原理：</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-c" data-lang="c">map(k1,v1) -&gt;list(k2,v2)
reduce(k2,list(v2)) -&gt;list(v2)</code></pre>
</div>
</div>
<div class="paragraph">
<p>可以借鉴其思想实现自己的 map reduce 框架：</p>
</div>
<div class="listingblock">
<div class="content">
<pre>https://github.com/pingcap/talent-plan/tree/master/tidb/mapreduce</pre>
</div>
</div>
<div class="paragraph">
<p>更详细的说明可以自行阅读 Google 的 map reduce 论文</p>
</div>
<div class="paragraph">
<p>map reduce 并不是分布式系统的专利：</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-shell" data-lang="shell">cat /var/log/nginx/access.log |
      awk '{print $7}' |
      sort             | <i class="conum" data-value="1"></i><b>(1)</b>
      uniq -c          |
      sort -r -n       |
      head -n 5</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>sort 工具就会进行外部排序，和 map reduce 过程非常相似。</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>map reduce api 过于底层，如今离线系统基本不需要再写 map reduce 脚本了。比如我们可以直接写 hive SQL。</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="true-hadoop-hive-lambda-kappa"><a class="anchor" href="#true-hadoop-hive-lambda-kappa"></a>在 hadoop/hive 生态上兴起的 lambda/kappa 架构</h2>
<div class="sectionbody">
<div class="imageblock">
<div class="content">
<img src="lambda-arch.png" alt="lambda arch.png">
</div>
<div class="title">Figure 1. lambda architecture</div>
</div>
<div class="paragraph">
<p>lambda 架构之前非常流行，甚至现在在我们这里也非常流行。问题也是显而易见的：</p>
</div>
<div class="ulist">
<ul>
<li>
<p>在线离线技术栈不同</p>
</li>
<li>
<p>两套代码，两倍人力</p>
</li>
<li>
<p>一个月开发，两个月对数据</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>twitter 尝试解决这个问题，推出了
<a href="https://github.com/twitter/summingbird">summingbird</a>
。一套代码编译出在线部分和离线部分。然而功能上只能支持在线/离线交集，且较为复杂，没有流行起来。</p>
</div>
<div class="imageblock">
<div class="content">
<img src="kappa-arch.png" alt="kappa arch.png">
</div>
<div class="title">Figure 2. kappa architecture</div>
</div>
<div class="paragraph">
<p>来自这里：
<a href="https://www.oreilly.com/ideas/questioning-the-lambda-architecture">Questioning the Lambda Architecture</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="true-spark"><a class="anchor" href="#true-spark"></a>半路杀出个 Spark</h2>
<div class="sectionbody">
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>在 2014 年 11 月 5 日举行的 Daytona Gray Sort 100TB Benchmark 竞赛中，Databricks 用构建于 206 个运算节点之上的 Spark 运算框架在 23 分钟内完成 100TB 数据的排序，一举击败了该赛事 2013 年的冠军— Yahoo 团队建立在 2100 个运算节点之上的 Hadoop MapReduce 集群，该集群耗时 72 分钟排序了 102.5TB 的数据。换句话说，Spark 用了十分之一的资源在三分之一的时间里完成了 Hadoop 做的事情。</p>
</div>
</blockquote>
<div class="attribution">
&#8212; internet
</div>
</div>
<div class="paragraph">
<p>Spark 大量使用内存而非磁盘来存储中间结果，比傻用磁盘的 hadoop 快几十倍是正常的。</p>
</div>
<div class="paragraph">
<p>除了性能优化之外，Spark 还做了一些抽象，这里面最重要的便是 RDD：</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>RDD(Resilient Distributed Datasets) ，弹性分布式数据集， 是分布式内存的一个抽象概念，RDD 提供了一种高度受限的共享内存模型，即 RDD 是只读的记录分区的集合，只能通过在其他 RDD 执行确定的转换操作（如 map、join 和 group by）而创建，然而这些限制使得实现容错的开销很低。对开发者而言，RDD 可以看作是 Spark 的一个对象，它本身运行于内存中，如读文件是一个 RDD，对文件计算是一个 RDD，结果集也是一个 RDD ，不同的分片、 数据之间的依赖 、key-value 类型的 map 数据都可以看做 RDD。</p>
</div>
</blockquote>
<div class="attribution">
&#8212; 百科
</div>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Spark 中每个 transform 的返回值都是 RDD，也就是 transform 是那些真正转换了RDD的操作，而 Action 操作会返回结果或把RDD数据写到存储系统中。Spark 在遇到 Transformations 操作时只会记录需要这样的操作，并不会去执行，需要等到有 Actions 操作的时候才会真正启动计算过程进行计算。</p>
</div>
</blockquote>
<div class="attribution">
&#8212; spark
</div>
</div>
<div class="paragraph">
<p>类似于函数式编程中的惰性求值。什么是惰性求值？举个例子：</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-c" data-lang="c">let rhs = rhs.replace("\'", "\"");
let r_vec: Vec&lt;&amp;str&gt; = rhs
    .trim_left_matches("(")
    .trim_right_matches(")")
    .split(",")
    .map(|v| v.trim())
    .collect();</code></pre>
</div>
</div>
<div class="paragraph">
<p>别看函数调用多，编译器甚至可以对中间某些可以进行合并的操作主动合并，去掉冗余操作，甚至可能比手写的过程式代码性能要好。</p>
</div>
<div class="paragraph">
<p><a href="http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIExamples.html">RDD 操作大全</a></p>
</div>
<div class="paragraph">
<p>鉴于 Spark 已经日薄西山，为了给自己减负，我们就可以不用学习了。开心。</p>
</div>
<div class="paragraph">
<p>Spark 没风光几年，就被新兴的流式计算框架降维打击了。</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="true-flink"><a class="anchor" href="#true-flink"></a>新兴流式计算框架 Flink</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Flink 在业务上解决了哪些痛点呢？来看看 Flink 官方文档中阐述的三种应用场景:</p>
</div>
<div class="sect2">
<h3 id="true-"><a class="anchor" href="#true-"></a>事件驱动型应用</h3>
<div class="imageblock">
<div class="content">
<img src="usecases-eventdrivenapps.png" alt="usecases eventdrivenapps.png">
</div>
<div class="title">Figure 3. event driven apps</div>
</div>
<div class="paragraph">
<p>如：</p>
</div>
<div class="ulist">
<ul>
<li>
<p>司机实时成交率</p>
</li>
<li>
<p>司机实时在线时长</p>
</li>
<li>
<p>司机实时反作弊</p>
</li>
<li>
<p>实时疲劳驾驶检查</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="true--2"><a class="anchor" href="#true--2"></a>数据分析应用</h3>
<div class="imageblock">
<div class="content">
<img src="usecases-analytics.png" alt="usecases analytics.png">
</div>
<div class="title">Figure 4. analytics</div>
</div>
<div class="paragraph">
<p>如：</p>
</div>
<div class="ulist">
<ul>
<li>
<p>实时 xx/yy/zz 大盘</p>
</li>
<li>
<p>司机实时组织化收入大盘</p>
</li>
<li>
<p>司机实时平均 iph</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="true--3"><a class="anchor" href="#true--3"></a>数据管道应用</h3>
<div class="imageblock">
<div class="content">
<img src="usecases-datapipelines.png" alt="usecases datapipelines.png">
</div>
<div class="title">Figure 5. data pipelines</div>
</div>
<div class="paragraph">
<p>如：</p>
</div>
<div class="ulist">
<ul>
<li>
<p>异步写入的 order feature system 中的所有订单特征</p>
</li>
<li>
<p>电子商务中的实时查询索引构建</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>可见本组拥有所有流式计算相关的业务场景。</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="true--4"><a class="anchor" href="#true--4"></a>流式计算的一些概念</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="truebounded-unbounded"><a class="anchor" href="#truebounded-unbounded"></a>bounded/unbounded</h3>
<div class="paragraph">
<p>有界，无界。在离线脚本开始运行时，可以认为某天的数据已经是完整的了，这便是“有界”。</p>
</div>
<div class="paragraph">
<p>而实际上业务并不是这样的。在司机实时成交率计算中，订单在判责结束后，可能会过很久(1-3天)，司机对结果进行申诉。申诉通过后，需要修改计算结果。纯粹的离线系统无法适应这样的场景。</p>
</div>
<div class="paragraph">
<p>源源不断流入的业务领域事件，往往是不会停止的。离线系统只是强行划界而已。</p>
</div>
</div>
<div class="sect2">
<h3 id="trueidempotent"><a class="anchor" href="#trueidempotent"></a>idempotent</h3>
<div class="paragraph">
<p>幂等性。如一个接口相同的多次调用会得到相同的计算结果。</p>
</div>
<div class="ulist">
<ul>
<li>
<p>incrby 是幂等的么？</p>
</li>
<li>
<p>update order set order_status = finished where order_id = 434453 是幂等的么</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="trueevent-time-vs-processing-time"><a class="anchor" href="#trueevent-time-vs-processing-time"></a>event time vs processing time</h3>
<div class="ulist">
<ul>
<li>
<p>event time : 事件实际发生的时间</p>
</li>
<li>
<p>processing time : 事件到达后端系统的时间</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>processing time 一般都是有序的(废话)。而 event time 则不一定。流式计算系统一定程度上解决了 event time 乱序的问题。通过什么来解决的呢？</p>
</div>
</div>
<div class="sect2">
<h3 id="truewindow-trigger-evictor"><a class="anchor" href="#truewindow-trigger-evictor"></a>window &amp;&amp; trigger &amp;&amp; evictor</h3>
<div class="imageblock">
<div class="content">
<img src="window.jpg" alt="window.jpg">
</div>
<div class="title">Figure 6. window</div>
</div>
<div class="paragraph">
<p>具体含义可以参考 <a href="https://blog.csdn.net/u4110122855/article/details/81360381">这里</a>。</p>
</div>
<div class="paragraph">
<p>在某个 window 结束之后，可以触发 trigger。</p>
</div>
<div class="paragraph">
<p>在Trigger触发之后，在窗口被处理之前，Evictor（如果有Evictor的话）会用来剔除窗口中不需要的元素，相当于一个filter。</p>
</div>
</div>
<div class="sect2">
<h3 id="truewatermark"><a class="anchor" href="#truewatermark"></a>watermark</h3>
<div class="paragraph">
<p>似乎在论文里叫 barrier&#8230;&#8203;</p>
</div>
<div class="imageblock">
<div class="content">
<img src="stream_watermark_in_order.svg" alt="stream watermark in order.svg">
</div>
<div class="title">Figure 7. 有序事件和 watermark</div>
</div>
<div class="imageblock">
<div class="content">
<img src="stream_watermark_out_of_order.svg" alt="stream watermark out of order.svg">
</div>
<div class="title">Figure 8. 乱序事件和 watermark</div>
</div>
<div class="imageblock">
<div class="content">
<img src="parallel_streams_watermarks.svg" alt="parallel streams watermarks.svg">
</div>
<div class="title">Figure 9. 并行数据流和 watermark</div>
</div>
<div class="paragraph">
<p>可以参考
<a href="https://blog.csdn.net/Jiny_li/article/details/86516762">这里</a>.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>window 和 watermark 是啥关系？</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="truelateness"><a class="anchor" href="#truelateness"></a>lateness</h2>
<div class="sectionbody">
<div class="paragraph">
<p>可能某些特定的元素会违背水印的条件，也就是说即使是Watermark(t)已经发生了，但是还会有许多时间戳t'&#8656;t的事件发生。事实上，在真实的设置中，某些元素可以任意延迟，因此指定一个时间，在这个时间内所有在一个特定事件时间戳的事件都会发生是不可能的。</p>
</div>
<div class="paragraph">
<p>延迟事件到达后可以选择丢弃，还是更新之前的结果。</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="trueexactly-once"><a class="anchor" href="#trueexactly-once"></a>exactly-once</h2>
<div class="sectionbody">
<div class="paragraph">
<p>有且只有一次。并不是那么简单的。</p>
</div>
<div class="paragraph">
<p><a href="https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">end-to-end exactly once</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="true--5"><a class="anchor" href="#true--5"></a>流式计算理论基石</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="truemillwheel-paper"><a class="anchor" href="#truemillwheel-paper"></a>millwheel paper</h3>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework’s fault-tolerance guarantees.</p>
</div>
</blockquote>
<div class="attribution">
&#8212; millwheel paper
</div>
</div>
<div class="paragraph">
<p>重点：用户指定有方向的计算图、每个节点的应用代码。系统负责状态持久化，和数据的流动，框架保证容错。</p>
</div>
</div>
<div class="sect2">
<h3 id="truedistributed-snapshot"><a class="anchor" href="#truedistributed-snapshot"></a>distributed snapshot</h3>
<div class="paragraph">
<p>leslie lamport 在 80 年代发表的论文，有很多不说人话的地方。主要解决的是一致性的问题。</p>
</div>
<div class="paragraph">
<p>简单来讲，如果在系统内部流动的数据是金额，并且没有外部转入，或者数据转出到外部。每次采集到的快照，能够保证全局总额是不变的。</p>
</div>
<div class="imageblock">
<div class="content">
<img src="lamport_snapshot.png" alt="lamport snapshot.png">
</div>
<div class="title">Figure 10. distributed snapshot</div>
</div>
<div class="paragraph">
<p>重点：所有节点和边均有状态。每个节点负责记录自己的状态，以及那些“入边”的状态。所谓的状态，其实就是某种值。</p>
</div>
<div class="paragraph">
<p>有理论支持，流式计算系统才能证明自己的快照从理论上来讲是“正确”的。</p>
</div>
</div>
<div class="sect2">
<h3 id="truelightweight-distributed-snapshot"><a class="anchor" href="#truelightweight-distributed-snapshot"></a>lightweight distributed snapshot</h3>
<div class="paragraph">
<p>对用户代码进行分析，判断计算图是否有环，分别采用两种算法注入 barrier 并采集全局的 snapshot。</p>
</div>
<div class="imageblock">
<div class="content">
<img src="flink_acyclic_snapshot.png" alt="flink acyclic snapshot.png">
</div>
<div class="title">Figure 11. 无环图快照算法</div>
</div>
<div class="imageblock">
<div class="content">
<img src="cyclic_snapshot-1.png" alt="cyclic snapshot 1.png">
</div>
<div class="title">Figure 12. 有环图快照算法</div>
</div>
</div>
<div class="sect2">
<h3 id="truethe-dataflow-model-paper"><a class="anchor" href="#truethe-dataflow-model-paper"></a>The Dataflow Model paper</h3>
<div class="paragraph">
<p>这一篇的内容在新书 《Streaming Systems》大多有讲，不用读这个。去读书就好。</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="true--6"><a class="anchor" href="#true--6"></a>流式计算领域的混战</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Spark vs Flink vs Beam。</p>
</div>
<div class="paragraph">
<p>Spark 的理论是，流处理是批处理的特殊情况。</p>
</div>
<div class="paragraph">
<p>Flink 的理论是，批处理是流处理的特殊情况。</p>
</div>
<div class="paragraph">
<p>Beam 的理论是，我全都要：</p>
</div>
<div class="imageblock">
<div class="content">
<img src="beam_architecture.png" alt="beam architecture.png">
</div>
<div class="title">Figure 13. Beam Architecture</div>
</div>
<div class="paragraph">
<p>支持多语言，再翻译成对应的执行任务：</p>
</div>
<div class="imageblock">
<div class="content">
<img src="beam-runtime.jpg" alt="beam runtime.jpg">
</div>
<div class="title">Figure 14. Beam Runtime</div>
</div>
<div class="paragraph">
<p>Pulsar vs Kafka。</p>
</div>
<div class="paragraph">
<p>没时间调研，就不展开说了。</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="true--7"><a class="anchor" href="#true--7"></a>课后问题</h2>
<div class="sectionbody">
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>流式计算框架是怎么保证最终一致的？</p>
</div>
</td>
</tr>
</table>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>window、watermark、trigger 是如何协作的</p>
</div>
</td>
</tr>
</table>
</div>
<div class="imageblock">
<div class="content">
<img src="coop.jpg" alt="coop.jpg">
</div>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
<div class="paragraph">
<p>window、watermark、trigger 是如何协作的</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div id="footer">
<div id="footer-text">
Last updated 2019-07-03 13:33:29 CST
</div>
</div>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.9.1/styles/github.min.css">
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.9.1/highlight.min.js"></script>
<script>hljs.initHighlighting()</script>
</body>
</html>